Crowdsourcing Ground Truth for Medical Relation Extraction
نویسندگان
چکیده
منابع مشابه
Crowdsourcing Ground Truth for Medical Relation Extraction
Cognitive computing systems require human labeled data for evaluation, and often for training. The standard practice used in gathering this data minimizes disagreement between annotators, and we have found this results in data that fails to account for the ambiguity inherent in language. We have proposed the CrowdTruth method for collecting ground truth through crowdsourcing, that reconsiders t...
متن کاملCrowdsourcing Ambiguity-Aware Ground Truth
The process of gathering ground truth data through human annotation is a major bottleneck in the use of information extraction methods. Crowdsourcing-based approaches are gaining popularity in the attempt to solve the issues related to volume of data and lack of annotators. Typically these practices use inter-annotator agreement as a measure of quality. However, this assumption often creates is...
متن کاملEmpirically-derived Methodology for Crowdsourcing Ground Truth
The main challenge for cognitive computing systems, and specifically for their natural language processing, video and image analysis components, is to be provided with large amounts of training and evaluation data. The traditional process for gathering ground truth data is lengthy, costly, and time consuming: (i) expert annotators are not always available; (ii) automated methods generate data w...
متن کاملIs Crowdsourcing for Optical Flow Ground Truth Generation Feasible?
In 2012, three new optical flow reference datasets have been published, two of them containing ground truth [1,2,3]. None of them contains ground truth for real-world, large-scale outdoor scenes with dynamically and independently moving objects. The reason is that no measurement devices exists to record such data with sufficiently high accuracy. Yet, ground truth is needed to assess the safety ...
متن کاملSelf-Crowdsourcing Training for Relation Extraction
One expensive step when defining crowdsourcing tasks is to define the examples and control questions for instructing the crowd workers. In this paper, we introduce a self-training strategy for crowdsourcing. The main idea is to use an automatic classifier, trained on weakly supervised data, to select examples associated with high confidence. These are used by our automatic agent to explain the ...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: ACM Transactions on Interactive Intelligent Systems
سال: 2018
ISSN: 2160-6455,2160-6463
DOI: 10.1145/3152889